The proliferation of networks and devices communicating through them has been accompanied by a proliferation of intentional misuse and disruption. Devices reachable through networks are frequently subjected to denial-of-service (DoS) attacks, brute force attacks, port scans, malware distribution attempts, SSL (secure socket layer) attacks, botnet attacks, URL (Universal Resource Locator) redirection attacks, address spoofing attacks, and others. Many forms of network intrusion and disruption have long been known, and new threats will continue to emerge.
Network intrusion detection (NID) systems have been used to detect and prevent network-based attacks. Centralized NID systems place intrusion detection functionality at key points of a network, such as edge routers and access routers. Some NID systems operate at the application layer and are deployed at end hosts/servers. NID systems can create bottlenecks and often involve costly hardware and expensive, high-capacity network links to handle large amounts of data. Storing and processing large amounts of data with minimal impact on network performance can require significant hardware resources. Some NID systems route network traffic through external servers that sandbox or divert malicious traffic. Such third-party services are costly and risk potential security compromises.
Centralized NID systems also suffer from a limited view of network activity. Because packets are inspected at a limited number of points in a network, some packet data might not be analyzed by an NID system. Transformations such as address translation, tunneling, encapsulation, and link encryption, can cause packet data to be apparent at some points of a network and opaque at other network locations where NID devices are operating. In other words, not all of the potentially threat-related payloads and header fields that are active on a network will be parseable at the network junctures where NID devices reside.
Common NID approaches have other disadvantages. For example, most NID systems use a set of known attributes or contents of packets or flows to identify threats. When a new type of threat or attack emerges, a human network administrator might notice a problem, laboriously gather clues from network traces, taps, host log files, router logs, etc., consult with administrators of other networks, and take time to isolate the network attack and identify its characteristics. A new threat profile or pattern might then be manually added to an NID system to address a repeat of a similar attack in the future. This cycle of identifying new threats and updating NID systems is expensive, time-consuming, and reactive/never ending. In addition, new threats can do significant harm before they are detected and mitigated. NID systems that detect threats by inspecting packets for pre-defined threat profiles or patterns inherently lag present conditions, e.g., zero-day attacks on a network. Moreover, small modifications to a particular known attack can render it undetectable as its pre-defined pattern or regular expression may no longer be valid. Administrators and their tools must constantly adapt to safeguard network security and performance.
Embodiments discussed below address one or more needs such as efficiently and automatically detecting and mitigating network attacks or anomalies, in near real time, on potentially extensive and busy networks, in a distributed scalable manner, without relying on a priori definitions or indicia of particular attacks.